CTMCs with Costs and Rewards

We consider continuous time Markov chains (CTMCs) where there are costs and rewards associated with each state. For such a CTMC the objective usually is to calculate either the long-run average costs incurred per unit time or the total discounted cost incurred over an infinite horizon. We illustrate the calculations using several examples and conclude with applications to system performance analysis. Consider a system that can be modeled as a CTMC {X(t), t ≥ 0} such that X(t) is the state of the system at time t. Note that X(t) could possibly be a vector but for notational convenience we sometimes write P{X(t) = j} with the understanding that j in those cases is also a vector. Let S be the state space of this CTMC, i.e. the set of all possible values of X(t) for all t. We use the notation Q to denote the infinitesimal generator matrix of the CTMC with negative values along the diagonals such that each row sums to zero. For terminologies and notation used for CTMCs refer to earlier articles of this encyclopedia. The reader is also encouraged to consult texts such as Kulkarni [1] and Ross [2]. For CTMCs with costs and rewards, an additional piece of notation is necessary. Define Ci as the cost incurred by the system per unit time it spends in state i, for all i ∈ S. Notice that if Ci is negative it is equivalent to the system getting a reward (as opposed to cost) of −Ci per unit time it spends in state i. Here we present only analysis of systems over an infinite horizon. In particular, we consider two infinite horizon cases: average costs and discounted costs. Each case has its own pros and cons that we will subsequently discuss along with some examples. Then we conclude by presenting applications to performance analysis.